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ABSTRACT 

A multi-State, quasi-experimental study was conducted as a longitudinal, two-year follow-up of 
participation in an online, on-demand professional development (PD) program. The purpose was 
to ascertain whether student gains were sustained in a second year of PD participation. Data 
verified gains in Year 1 versus Pre-PD baseline, with continued gains in Year 2 atop those 
achieved in year 1 of PD participation, reflecting a positive trend and continued advantage over 
non-PD schools in the same districts. Results showed that student in PD schools gained 7.7% 
(p<.01) more in Math in year 2 atop 18.9% (p<.001) gains from year 1, versus gains of 0.5% (ns) 
and 4.2% (p<.01) for non-PD schools in the same districts. Similarly, students in the PD schools 
gained 10.2% (p<.01) more in Reading in year 2 atop 18.9% (p<.001) gains from year 1, versus 
gains of 0.5% (ns) and 4.2% (p<.01) for non-PD schools in the same districts. Total gains from 
baseline for PD schools were 28.1% (p<0.001) in Math for PD schools versus 4.7% (p<0.01) for 
the same districts collectively, and 30.2% (p<0.001) in Reading for PD schools versus 6.0% 
(p<0.01) for the same districts collectively. Findings support the higli-participation use of 
Internet-based, on-demand professional learning for improving teacher effectiveness and 
sustained impacts on student performance. 
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INTRODUCTION 



he main purpose of professional development (PD) remains primarily to improve the impact teachers 
have on their students (Borko, 2004; Fishman et.al., 2003; Guskey, 2002; Loucks-Horsley et.al., 
2009; Mizell, 2010; Shaha et.al., 2004; Villegas-Reimers, 2003; Wilson & Berne, 1999). Many PD 
approaches provide teachers with an opportunity for time off from the classroom routine to participate in PD- 
focused workshop and seminar settings. The advent of video-enabled PD approaches began to provide educators the 
ability to obtain access to PD in smaller portions without costly absences from day-to-day classroom settings 
(Colestock & Sherin, 2009; Ingvarson et al. 2005; Sherin & van Es, 2005, 2009). Avoiding removal of teachers for 
PD thus reduced the cost of PD in terms of teacher replacement, but also provided needed or wanted PD training 
with immediate applicability. 


More recently, advances in technology have enabled teachers to access PD through online, on-demand, and 
Internet based offerings. These offerings have evolved to reflect our increased understanding of how teachers can 
best learn about teaching, and teaching better, have advanced (Bahr et al., 2004; Borko, 2004; Ingvarson et al. 2005; 
Putnam & Borko, 2000). They also make possible levels of selectivity in teacher participation enabling more 
personally relevant selections and participation (Shaha et al. 2014). As a result, the expectations of PD offerings 
have progressed (Sherin & Han, 2004; Sherin & van Es, 2005, 2009). 


Regardless of improved understanding of how teachers learn, and better Internet-based delivery options, 
studies quantifying and validating the impact of PD on student performance have remained rare at best (c.f. Shaha 
et.al., 2004). Masters, et.al., commented specifically on this “dearth of scientific research’’ (Masters et al. 2012). 
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Such is a travesty, given the pressure on administrators and educators to improve the impact teachers have on 
students. Decisions regarding PD expenditures and participation remain based primarily on marketing and 
reputation, and not on statistically rigorous, data-rich studies. However, as the costs associated with providing 
educational offerings are more rigorously scrutinized into the future, the need to prove PD impact will become 
increasingly important (Farnsworth et al. 2002; Lewis et al. 2003; Magidin et al. 2012; Rienties et al. 2013). 
Substantially more research is needed to establish quantitatively a compelling link between teacher participation in 
PD and gains in student performance (Garet et al. 2001; Desimone et al. 2002; Shaha et al. 2004; Ingvarson et al., 
2005; Buczynski & Hansen 2010; Avalos 2011). 

In the interest of verifying the impact of PD participation, two important trends are recently materializing. 
First, a handful of studies have emerged that add substance to arguments that applying rigorous program evaluation 
designs and techniques to PD offerings is increasingly necessary and valid to meet the needs for increased 
accountability for ensuring that PD expenditures reflect investments (c.f. Shaha et. al. 2004; Shaha & Ellsworth, 
2013). Second, additional recent studies have shown that quantifying improvements in student performance on 
standardized tests provides a legitimate, rigorous and generalizable approach to validating value (Buczynski and 
Hansen 2010; Avalos 2011; Garet et al. 2001; Desimone et al. 2002; Ingvarson et al. 2005; Wasik & Hindman 
2011). Thus the measure of teacher impact on students as enhanced through PD participation has leveraged scores 
as a useful metric for assessing and quantifying impact. 

Reflecting the evolution of PD, studies related to PD delivered online and on-demand have recently 
increased (c.f. Shaha & Ellsworth 2013). Studies have logically verified that the level of teacher activity and 
engagement with online PD affects the correlated impact achieved for student performance - students of teachers 
with highly active engagement in online PD outpaced the students of teachers reflecting more passive PD 
participation, such as viewing videos alone, versus downloading and uploading material (Garet et al. 2001; 
Desimone et al. 2002; King 2002; Darling-Hammond 2004; Santagata, 2009; Shaha & Ellsworth 2013). 

The purpose of this research was to perform a longitudinal, two-year follow-up study leveraging a multi- 
State, quasi-experimental design. The study will reflect the comparative impact of PD on the students of 
participating teachers versus comparable, non-participating schools within same districts. We built upon the 
findings of a one-year, pre-versus-post, quasi-experimental study formerly published and conducted a second year 
follow-up (Shaha & Ellsworth, 2013). Our goal was to determine whether advances experienced for students in 
their second year of PD participation experienced similar gains as were quantified in the first pre-versus-post year, 
or whether those early gains diminished or were improved upon. 

METHODS 

The study contrasted gains in performance for schools whose teachers participated in the online, on- 
demand PD offering (hereafter schools) versus the remaining schools in the same district whose teachers did NOT 
participate (hereafter districts). Additionally, this study was an extension of a previously executed pre-versus-post 
study, and involved the addition of a second post-implementation year of data to the previously analyzed data 
(Shaha & Ellsworth, 2013). The schools and districts included in analyses are the same with the additional year of 
data added for examining sustainment. 

The study design reflected a quasi-experimental approach (Cook & Campbell 1979): Pre-PD levels of 
student performance were compared with Year 1 and Year 2 post-implementation. By design, performance in PD- 
participating schools was contrasted with that for their own corresponding districts, including mainly non¬ 
participating schools. Contrasting participating schools with their own districts controlled for socio-economic and 
demographic differences. Thus findings would not be attributed to demographic or socio-economic differences 
between groups, or even to general capabilities of either teachers or students - schools and districts represent 
interchangeable groups in a design. 

School participation was defined as use of a single commercially available online, on-demand PD product 
widely used in the United States (PD 360 ® and Observation 360 ®, School Improvement Network, Salt Lake City). 
The enabled teachers to participate in a full range activities which range instructional videos on teaching techniques. 
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to communities of other users, to posting and downloading PD-related materials. Inclusion criteria for schools was 
defined as a minimum average of 90 minutes per teacher of logged-in time cumulatively school-wide - this 
minimum adopted to ensure sufficient PD use to enable that conclusions be drawn regarding the impact of PD use 
beyond mere sign-on alone or minimal usage. Comparison Further, schools were included along with their 
corresponding districts only if fewer than 25% of schools within a given district were categorized as PD participants 
and met inclusion criteria. 

PD school participation data analyzed were extracted from the data automatically captured by the PD 
provider (School Improvement Network) as a result of PD use, and corresponding district data from the same 
sources, including verification off eligibility for district meeting non-PD participation criteria. The pre-PD included 
data from the 2009-2010 school year, and the two post-implementation years - 2010-11 and 2011-12 - were the 
corresponding data from the subsequent two school years thereafter. 

Student performance data were gathered from publically available web sources for the three consecutive 
school years. Any data not available online were gathered by telephone requests, all with Institutional Review 
Board (IRB) approval where such existed. For analyses, student performance data were defined as the sum of the 
percentages of students rated either Proficient or Advanced on the respective standardized student performance tests 
for each school - no changes in testing were noted within the sample for the three consecutive years of data. 
Schools and districts included were engaged in performance tests for which Proficient and Advanced were common 
labels or levels of achievement above other levels indicating performance at grade level or below for that assessment 
instrument. District data could not be recomputed with exclusion of participating schools, and therefore some 
changes in performance for the collective districts do reflect changes in performance for the participating schools 
within. 


The final data set included the same 169 schools in 73 districts and 19 States for the third year of data as 
was analyzed in the previous 2-year pre-versus-post study. All percent changes reported were calculated as net 
change in performance, divided by performance for the earlier of the two years: e.g. (Year 2 - Year 1)/ Year 1. 
Effect size was computed using Cohen's f~ (Cohen, 1992). Significance for percent change figures was determined 
by contrasting versus zero change (0.0%) by a t-Test for proportions (Rice, 2006). All analyses were conducted 
using SPSS version 17.0 or higher (PASW Statistics, SPSS, 2009, with SAS for confirmatory purposes when results 
were close to p<0.05). Minimum level of statistical significance was determined a priori at p<0.05. 

RESULTS 


Mathematics 

Year 1 of PD Participation 

Student performance levels were contrasted for schools versus districts, for pre-PD versus Year 1 post-PD 
(See Table 1, Figures 1, Shaha & Ellsworth, 2013). For Math, results showed that students of teachers in PD- 
inclusion schools experienced a net change of 11.1 percentage points in students classified as Proficient or 
Advanced versus a 2.6 net change for districts. That difference within schools equated to an 18.9% (p<0.001) 
increases in performance year-over-year for PD participation, versus 4.2% (p<0.01) comparative increase for 
districts. The percent change within PD schools was 4.2 times greater than for their respective districts. The effect 
size for schools was 0.54, which is 4.15 times greater than the effect size of 0.13 for the districts. Performance 
within district was marginally better than for schools in the pre-PD year (p=0.065), and that pattern was reversed for 
Year 1 post implementation wherein schools were marginally better (p=0.085). 
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Math 


Table 1. Student Math performance for participating schools versus their districts 




Pre 



Year 1 





Advanced + 
Proficient 

Advanced + 
Proficient 

Net change vs. Pre 

Effect size 

Pet Change vs. Pre 

Significance 

Schools 

Mean 

58.4 

69.5 

11.1 

4.2 

18.9% 

p<0.001 


StDev 

21.9 

21.1 





Districts 

Mean 

62.9 

65.5 

2.6 


4.2% 

p<0.001 


StDev 

20.4 

20.2 





Comparative Net Difference 


-4.5 

4.1 





Significance 


p=0.065 

p=4.1 





Net Difference in Change 




8.4 


14.7% 


Significance 




p<0.001 


p<0.001 



Year 2 



Advanced + 
Proficient 

Net Change 
vs. Year 1 

Effect Size 

Pet Change 
vs. Year 1 

Significance 

Net Change 
vs. Pre 

Effect Size 

Pet Change 
vs. Pre 

Significance 

Schools 

Mean 

74.8 

5.3 

15.0 

7.7% 

P<0.01 

16.4 

5.6 

28.1% 

p<0.001 


StDev 

23.4 









Districts 

Mean 

65.8 

0.4 


0.5% 

ns 

3.0 


4.7% 

p<0.001 


StDev 

22.3 









Comparative Net Difference 


9.0 





13.5 




Significance 


p<0.001 





p<0.001 




Net Difference in Change 



500 




13.5 




Significance 



p<0.01 




p<0.001 
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Year 2 of PD Participation 

Student performance was contrasted for schools versus districts for gains Post-PD between Year 1 and Year 
2. Students of teachers in PD-inclusion schools experienced a second-year net performance improvement of 5.3 
percentage points, versus a 0.4 net change for their respective districts. That equaled a 7.7% (p<0.01) increase in 
performance year-over-year for schools, versus 0.5% increase for the districts (not significant, hereafter “ns”). The 
improvement within schools was 15.5 times greater than for their districts. The effect size for schools was 0.24, 
over 17 times greater than the 0.01 effect size for the districts. Performance within schools maintained and 
increased the advantage established in Year 1 over their district. 

Two Years of PD Participation 

Finally, results showed that students in the school group experienced a net performance improvement of 
16.4 percentage points when contrasting Year 2 with the pre-PD baseline, versus a 3.0 net change for districts. That 
equaled a 28.1% (p<0.001) increase in performance for schools by Year 2 over baseline, versus 4.7% (p<0.01) 
increase for their respective districts. The growth for students in PD schools was 5.6 times greater than for their 
respective districts. The two-year effect size for schools was 0.74, 5.29 times greater than the 0.14 two-year effect 
size for the districts. 

Reading 

Year 1 of PD Participation 

Contrasts were performed for student school performance levels for pre-PD versus Year 1 post-PD versus 
districts (See Table 1, Figures 1, Shaha & Ellsworth, 2013). Results showed that students in PD schools 
experienced a net change of 10.3 percentage points in Proficient plus Advanced versus a 1.6 net change for the 
corresponding districts. That difference within schools equated to an 18.1% (p<0.001) increase in performance 
year-over-year for PD participation, versus 2.6% (p<0.01) comparative increase for their districts. The percent 
change within PD schools was 6.4 times greater than for their respective districts. The effect size for schools was 
0.45, 6.34 times greater than the effect size of 0.07 for the districts. Performance within district was marginally 
better than for schools in the pre-PD year (p=0.079), and that pattern was reversed for Year 1 post implementation 
wherein schools were marginally better (p=0.063). 
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Table 2. Student Reading performance for participating schools versus their districts 


Reading 




Pre 



Year 1 





Advanced + 
Proficient 

Advanced + 
Proficient 

Net Change vs. 

Pre 

Effect Size 

Pet Change vs. 

Pre 

Significance 

Schools 

Mean 

56.9 

67.2 

10.3 

6.4 

18.1% 

pcO.OOl 


StDev 

22.4 

22.1 





Districts 

Mean 

61.2 

62.7 

1.6 


2.6% 

p<0.01 


StDev 

21.1 

20.7 





Comparative Net Dif 


-4.3 

4.5 





Significance 


p=0.079 

p=0.063 





Net Difference in Change 




8.7% 


15.5% 


Significance 




p<0.001 


pcO.OOl 



Year 2 



Advanced + 
Proficient 

Net Change 
vs. Year 1 

Effect Size 

Pet Change 
vs. Year 1 

Significance 

Net Change Effect size 
vs. Pre 

Pet Change 
vs. Pre 

Significance 

Schools 

Mean 

74.1 

6.9 

3.2 

10.2% 

pcO.OOl 

17.2 4.7 

30.2% 

pcO.OOl 


StDev 

19.9 








Districts 

Mean 

64.9 

2.2 


3.5% 

pcO.Ol 

3.7 

6.0% 

pcO.Ol 


StDev 

17.4 








Comparative Net Dif 
Significance 


9.2 

pcO.OOl 





13.5 

pcO.OOl 



Net Difference in Change 



4.7 




13.5 



Significance 



pcO.Ol 




pcO.OOl 
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Figure 2. Comparative pre versus post performance for Reading proficient and advanced 



50.0 

Pre Year 1 Year 2 


Year 2 of PD Participation 

Student performance was next contrasts for Year 1 and Year 2 of PD participation. Students in PD- 
participation schools experienced a second-year net performance improvement of 6.9 percentage points over Year 1, 
versus a 2.2 net change for their respective districts. That equaled a 10.2% (p<0.001) increase in performance year- 
over-year for schools, versus 3.5% increase for the districts (p<0.01). The improvement within schools was 3.2 
times greater than for their districts. The effect size for schools was 0.32, 2.91 times greater than the 0.11 effect size 
for the districts. 

Two Years of PD Participation 

Lastly, results showed that students in the school group experienced a net performance improvement of 
17.2 percentage by Year 2 versus the pre-PD baseline, in contrast with a 3.7 net change for their districts. That 
equaled a 30.2% (p<0.001) increase in performance for schools by Year 2 over baseline, versus 6.0% (p<0.01) for 
their districts. The growth for students in PD schools was 4.7 times greater than for their respective districts. The 
two-year effect size for schools was 0.81, 4.26 times greater than the 0.19 two-year effect size for the districts. 

Asymptotic Pattern 

The graphics depicting the findings for both Mathematics and Reading each portray an important, 
corroborative, asymptotic pattern. While growth continues in the second year of data, the amount of year-over-year 
gain decreased versus that achieved in the first year. That continued growth, curving slightly downward year-over- 
year, represents the ceiling effect 100% performance represents, wherein schools may incrementally get every-closer 
to that goal, but cannot exceed it, or possibly achieve it. Statistically, an asymptotic pattern is expected: Levels of 
students classified as Proficient plus Advanced cannot not exceed 100%, although that limit can be approached as an 
ultimate goal. 


DISCUSSION AND CONCLUSIONS 

Past research, although limited in volume and rigor, has recently begun to provide some basis for 
understanding the impact of PD its first year of pre-versus-post implementation. While even less research is 
available regarding online, on-demand professional development for pre-versus-post interpretation, the body of 
studies is growing. Despite those advances, still very studies few represent multi-State, high-sample-size analyses 
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quantifying PD impact. In that context, still woefully fewer studies exist offering substantive quantitative evidence 
of the impact of PD into subsequent years as substantiation of sustainment. 

We believe this may be the first such study. This multi-state, quasi-experimental study provides quantitate, 
compelling evidence of the impact of on-demand, Internet-based PD for enhancing teacher efficacy as measured by 
changes in student performance on standardized tests. First-year post-implementation gains showed that 
participation in the online, on-demand PD offering resulted in 4.5 times and 6.1 times greater gains for math and 
reading, respectively, for participating schools versus their respective, comparable district populations. In the 
second year, advantages were sustained with 5.3 times and 3.2 times greater gains for participating schools versus 
their district peers. Thus favorable trends persisted, and continued improvements were sustained. 

As with all quasi-experimental research, interpretation of these findings involves some extrapolation 
beyond the ideal controlled conditions of strict experimental research. It remains nearly impossible in education to 
study comparative student gains within environments with rigorous control-versus-treatment conditions. Therefore 
it will also remain challenging to firmly establish cause-and-effect conclusions based on the quasi-experimental 
design and naturalistic settings in which this study was achieved. However, the evidence remains compelling. 

Thus this may stand as the first-ever two-year, large sample, follow-up study quantifying the sustained 
impact of online, on-demand PD on student performance. It surely is one of very, very few studies establishing the 
sustained effect of any systematic PD on student performance contrasting comparable student populations in a large 
sample, multi-State design. Bottom line the data clearly and substantively indicate that continued PD participation 
results in sustained gains, and even continued additional gains, as teachers apparently leverage what they learn. 
Whether this is explained by acquisition of knowledge and skills from PD participation, or simply improved 
attention and focus on the part of classroom teachers, will remain unclear. Yet establishing such an explanatory 
nuance is likely less important than the formulaic conclusion that sustained participation in PD across years is 
correlated with sustained improvements in student performance. 
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